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Abstract. We investigate the retrieval phase diagrams of an asynchronous fully- 
connected attractor network with non-monotonic transfer function by means of a 
mean-field approximation. We find for the noiseless zero-temperature case that 
this non-monotonic Hopfield network can store more patterns than a network with 
monotonic transfer function investigated by Amit et al. Properties of retrieval phase 
diagrams of non-monotonic networks agree with the results obtained by Nishimori 
and Opris who treated synchronous networks. We also investigate the optimal 
storage capacity of the non-monotonic Hopfield model with state-dependent synaptic 
couplings introduced by Zertuche et al. We show that the non-monotonic Hopfield 
model with state-dependent synapses stores more patterns than the conventional 
Hopfield model. Our formulation can be easily extended to a general transfer 
function. 
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1. Introduction 

Statistical mechanical approaches were successful for investigation of equilibrium 
properties of associative memories or attractor networks. The Hopfield model 
which updates its state asynchronously was investigated from statistical mechanical 
point of view by Amit et al |§1 [Q , and a lot of interesting features were found. One 
of the main issues about the Hopfield model as an associative memory device is the 
critical storage capacity. Amit et al showed that the Hebbian learning in the 
Hopfield model leads to the optimal storage capacity ac = p/N = 0.138, where p 
is the number of embedded patterns and N is the number of neurons. Fontanari 
and Koberle extended the method of Amit et al to the synchronous networks and 
showed that the capacity remains the same ac — 0.138 and derived finite-temperature 
properties of synchronous networks. On the other hand, we can not obtain information 
on the dynamical process of retrieval by equilibrium statistical mechanics. 

Amari and Maginu Q proposed a signal-to-noise ratio analysis to investigate the 
dynamical properties of synchronous networks. They divided time-dependent local 
field h\ = (l/N) J^j ^i^j'^ji^) i^^o ^ signal part m^, which corresponds to the first 
{fj, = 1) term in the summation over fi, and a noise contribution corresponding 
to the rest (/it^l). They assumed that the time-dependent noise term obeys the 
Gaussian distribution during the dynamical processes and showed that the capacity 
is ttc — 0.159. Nishimori and Ozeki pointed out by Monte-Carlo simulations 
that the assumption of a Gaussian distribution of the noise term is valid at least 
within statistical uncertainties if the final retrieval is successful. And they extended 
the Amari-Maginu theory to the network which updates its state stochastically and 
investigated the properties of the Hopfield network at finite temperatures. The phase 
diagram obtained as the equilibrium limit of the extended Amari-Maginu dynamics is 
very similar to the phase diagram of Amit et al |^ . 

The limitation of storing patterns in Hopfield networks comes mainly from the 
Hebbian interactions = (V^) In fact, Gardner § § showed, 

in her pioneering papers, that the optimal storage capacity ac is 2 for the general 
interaction Jy . Many attempts have been made to increase the storage capacity of 
the Hopfield model to Gardner's limit ac = 2 by taking more complex synapses. 
Recently Zertuche et al ||ll| studied the storage capacity of the Hopfield model with 
state-dependent synapses by introducing a threshold parameter 77. This parameter 77 
determines which patterns contribute to the synapses. This synapse can be written as 
•^ij — (1/-^) J2fi ~ v'^/^)- In their model, only patterns whose correlation 

with the state of the networks is greater or equal to the threshold are left finite to give 
a Hebbian contribution to the synapses. The capacity of the Hopfield network with 
this type of synapses is found to increase ac from 0.138 to 0.171 at T = and 77 = 1.0. 

Nishimori and Opris ||l^ investigated the retrieval properties of an associative 
memory with a general transfer function using the extended Amari-Maginu theory Q . 
They obtained the optimal storge capacity for the non-monotonic transfer function by 
taking the equilibrium limit of the recursion relation of the Amari-Maginu dynamics 
and showed that networks with non-monotonic transfer functions yield an enhanced 
memory capacity than the conventional monotonic relation. This property of non- 
monotonic neural networks was also pointed out by Morita et al by Monte-Carlo 
simulation before Nishimori and Opris p^ . The reason why the optimal storage 
capacity of non-monotonic transfer function increases is that a weak value of the total 
input to a neuron implies a confused state and an inverted output for a weak input 
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might works as a trial toward an improved retrieval. 

In this paper we investigate the retrieval phase diagram of Hopfield networks which 
update asynchronously and have a non-monotonic transfer function by a mean-field 
theory of statistical mechanics proposed by Geszti | 
formulation of the mean-field approximation to the asynchronous Hopfield networks 
with a non-monotonic transfer function and equation of state are derived. In section 
3 we extend our formulation to the networks with a general type of transfer function. 
In section 4 we study the performance of non-monotonic Hopfield networks when their 
synapses depend on the state of networks using the method proposed by Zertuche et al 

. In section 5 we compare the results of our calculations with the results obtained 
by Nishimori and Opris |p^ . 



Ml 1 13 . In section 2 we show the 



2. Equations of state 

Most of the investigations which discussed equilibrium properties of fully-connected 
Hopfield networks by statistical mechanics were restricted to networks with 
equilibrium free energy. In order to show the existence of such a free energy, we must 
make it sure that the synaptic couplings are symmetric and the transfer function 
is monotonic. Our model in this paper has symmetric couplings, but the transfer 
function is non-monotonic. In order to overcome this difficulty we use the mean-field 
approximation as follows. 

Let us suppose that the ith neuron updates its state according to the next 
probability 

Prob(a,(< + 1)) = i [1 + a.it + l)f{hl)] (1) 
where cr,; = ±1, = ±1, and the local field to the ith neuron ft,* is defined as 

Then we can calculate the average value of cr,:(t + 1) as 

< a.,it + 1) >= (+l)xi [1 + f{hl)] + (-l)xi [1 - /(ft*)] 
= fitA) (3) 
For the equilibrium state we obtain the equation of state by mean field approximation 



<-^>=/(|i^EEer^;<-.>j (4) 

In this section we choose the function f{x) as 

f{x) = tanh {-/3{x + a)) + tanh {-f3{x - a)) + tanh (5) 



where a is a positive constant and /3 is a parameter related to the synaptic noise. 
This non-monotonic transfer function reduces to the form in Figure 1 in the limit 
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(3 = 1/T^oo. Then equation (4) can be rewritten explicitly as 

13 



< (Ji >~ tanh 



N 



3 M 



tanh 



tanh 



N 



l-i- 



EE?r^;<-.> 



3 M 



(6) 



We introduce the overlap between the equilibrium state of the network < <Ji > and 
an embedded pattern v as follows 



< CTi > 



(7) 



Using this overlap parameter, we may rewrite (6) as 



X < tanh 



tanh 



tanh 



/^Ee 



We divide the term ^i^^^L appearing above into three parts: the first for = \{^v) 
which corresponds to the retrieved state, the second corresponding to the term fi = v, 
and the rest. Then we get 



l^Cre.Hanh 



l^C'^e.Hanh 

i 

l^^re!tanh 



(9) 



Here, mi is of order 1, and the summation X^/^^i^i i/^fCi^'^M also of order 1, while 
the term S^Qrai, is much smaller, ©(I/a/ZV). Then we may expand the function tanh 
appearing in (9) to first order of ^Im,^. The terms X^^^i i, appearing in tanh 

may be regarded as Gaussian variables with mean zero and variance v "^/j^=Q;r 

p^ . Under this approximation, we may replace the summation i^/N)'^^ by a 
Gaussian integral. We next square the equation (9) to calculate r = X^i^^iti "f^v^ I ol- 
Following the procedure introduced by Gesti , we obtain the equations of state in 
the limit of N^oo as follows 



r=[l-Piq+ + q--q-l)Y 



(10) 



and 



m = — J Z?ztanh[/3(m + ^/arz — a)] 
— J Dzta,nh.[l3{m + ^farz + a)] 

+ J Dzta.nh[P{m + y/arz)] (11) 
where we set mi — m and introduced the Edward- Anderson [ p^ Uke order parameters 
q= J Dztanh^ [l3{mi + y/arz)] (12) 

1±= J Dztanh^ [(3{m + ^farz ± a)] (13) 
In the zero temperature hmit, these order parameters can be rewritten as 
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'? = i-\/r7r^exp(-— ) (14) 



'/±-l-A/z7^exp(-A^) (15) 



Then equations of state lead to 



, 2 f / (m + a)2\ / (m , 
1 + W <^ cxp + exp '— + exp 



and 



where 



Trar 1 V 2ar / \ 2ar / \ 2ar 



m = -erf ^ ^'"M + erf (17) 




erfc(a;)=l 



2 

erf(x) = ^ / rftexp(-i2) (18) 

For a— >c» one recovers the equations of state of the Hopfield model at T = obtained 
by Amit et al ||||4|. 

The neural network is useful as an associative memory as long as the mean field 
equations (10) and (11) have a solution of the form = (m,0, • • -,0,0) with m^O. 
At r = 0, there exist metastable states which are highly correlated with particular 
embedded patterns as long as p = aN < UcN. We solved the equation (16) and 
(17) numerically and obtained the critical capacity ac- This result is plotted as a 
function of the parameter a in figure 2. In this phase diagram, the region R denotes 
the retrieval phase and the region N/R means the non-retrieval phase where the self- 
consistent equations (16) and (17) do not have a non-zero solution of m. From this 
result, we see that Hopfield networks with non-monotonic transfer function store more 
patterns than the networks with monotonic transfer function |^ : Uc has the maximum 
value 0.211 at a = 1.77. The shape of the ada) curve has similar properties, in the 
following sense, with that of Nishimori and Opris who calculated this critical curve 
by the equilibrium relation of Amari-Maginu dynamics for synchronous networks: 
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• There exists a certain value of a that maximizes ac- 

• Qfc approaches the monotonic value (0.138 in the present case) in the limit a— >cx). 

An interesting observation is that an iterative solution of the self-consistent equations 
(16) and (17) showed oscillatory behavior in a restricted region around a = and 
a = in the phase diagram. In consideration of similar observations from dynamical 
treatments [|l^, such a phase should be characterized by time development of the 
overlap mt in the sense that mt+i > if mt < and mt+i < if mt > 0. However, it 
should be warned that as we treat the static properties of the non-monotonic Hopfield 
model, we can not extract the dynamical behavior of mt from our formulation, strictly 
speaking. 

Next, in order to get the a-T phase diagram, we solved the finite-temperature self 
consistent equations (10) and (11). For the case of a = 1.80 we plotted the a-T curve 
in figure 3. We have found that the transition from the (normal) retrieval phase to 
the spin glass phase is of first order. 



3. Extension to a general transfer function 

In this section wc show that our formulation can be extended to the networks with a 
general transfer function jl^ . The mean field equation of state for a general transfer 
function / is already given in (4). Equation (9) of the parameter can be rewritten 
for this general transfer function as follows. 

"^'' = ^E^^''/(E^r"^M) (19) 

i ^ 

Here we have included the effect of the control parameter /3, corresponding to the 
thermal noise, in the general function /. Wc should not forget that the absolute value 
of the function / does not exceed 1 because otherwise probabilistic interpretation (1) 
does not make sense. For the general transfer function /, we can obtain the equations 
of state in the same way as in the previous section. The result is 



(20) 



and 



where 



TO = y" Dz f{m + V^z) (21) 
Qo=y Dzf{m + V^z) (22) 



Q= J Dzf{m + ^/^z) (23) 

Note that by setting /(x) = tanh(/3a;) we recover the result by Amit e< aZ ||^. 

We next show that our equations of state for the general transfer function are 
different from the result of Nishimori and Opris jl^. They calculated the recursion 
relations of macro-variables mt and at (the latter being the measure of disturbance 
from non-retrieved patterns) by generalizing the Amari-Maginu type signal-to-noise 
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ratio analysis |^ to stochastic dynamical process in the case of synchronous dynamics. 
Their result is 

TOt+i = J Dzf{mt + (Ttz) (24) 

(Tt^i = a + 2amtmt+ih{mt, at) + afh^{mt,at) (25) 
where h is defined by 

h{mt,at)= Dzf'{mt + (Ttz) (26) 



Taking the equilibrium limit i— s-oo and setting doo — \/c(r and moo = 'ti, we get 
equations of state with respect to m and r as 

m = y Dzf{in + \focrz) (27) 
f + am^h{m, ^/ar) 



1 — {h{m, Y^ar)}^ 



(28) 



where 



M„>,vS^)^/i5=/'(m + v^=) (29) 

This is different from our result (20) and (2f). We may suppose that this difference 
comes from the difference between synchronous and asynchronous dynamics. There 
is no a priori reason why the equilibrium properties of synchronous networks should 
coincide with those with asynchronous networks. 



4. State-dependent synapses 

As long as the number of embedded patterns satisfies p<tiN, the noise term 
1/ Ci^Cf "^/j appearing in the mean-field equation is of order 0{1/N) and we 
can neglect this term. However, if p = aN with a finite, this same term becomes 
0(f) and this contribution can not be neglected. These non-retrieved memories 
{fiy^l,iy), which appear in u^i^i''^fj-' prevent networks from retrieving the 

embedded pattern. The storage capacity of the Hopfield model is limited by the 
contribution of a large number of weakly correlated patterns. For the conventional 
Hopfield model, patterns {^'^} are stored by the Hebbian type synaptic interaction 
•^ij = (1/-^) CfCj'- Therefore, in order to exclude non-retrieved memories which 
have small overlaps with the state of the network, we should modify the synaptic 
interaction so that only patterns with large overlaps with state contribute to the 
Hebbian rule ||Tf[ . Our main interest in this section is to what degree the stability of the 
memorized states is improved by this state-dependent synaptic interaction [jll] and 
how many patterns are stored in the Hopfield networks with non-monotonic transfer 
function. 

We use the next state-dependent synapses by introducing a threshold ri{ >0 ) 

^^^-^E^"^"©(V-7^) (30) 
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The factor of the step function Q{x) means that an embedded pattern is excluded 
if the overlap between the pattern and network state is below a threshold 
nT'n'^ < rf^ /N . We expect that the performance of a network as an associative memory 
is improved by introducing this type of synapses with threshold r/ > to exclude the 
spurious memories disturbing retrieval. We introduced the factor because is 
of order 1 / ViV. It is important to bear in mind that the conventional Hebb interaction 
Jij = is recovered by setting 77 = 0. Using this coupling, we rewrite 

the mean field equation obtained in section 2 as follows. 

i 

+ /3m,e(m,2 _ ^[1 _ tanh2/3(m + Tyf )] 

i 

- l^Y.^u^'^^m^+v': - a)] 

i 

- (3m^e{m^^ - ^[1 - tanh2/3(TO + ryf - a)] 

i 

i 

- f3m,e{m,^ - ^[1 - tanh'/3(m + yyf + a)] (31) 

i 

where we introduced as follows. 

E Ce,^-M0(-.' " 7^) (32) 

This is the sum of a large number {— aN) of small terms (of order 1/A^). We now 
assume that the small contributions (a't^I) have identical Gaussian distributions 
centered at zero, with variance cr^/iV. Strictly speaking, this statement is not exact 
because are related through (31). Nevertheless we accept this approximation in 
this paper. By the same arguments, Cj is assumed to have a Gaussian distribution 
with variance ar and average zero, so that 

ar = «(77r)'» (33) 
We also introduce the Edwards- Anderson like order parameters [|l6| as 

q = E tanh^/3(TO + Tyf ) = J Dz tanh^/3(m + y/arz) (34) 

i 

q± = — E] tanh^/3(m + ??f ±a) ~ J Dz tanh^/3(TO -|- -y/arz±a) (35) 

i 

Using these parameters, (31) leads to 

l-/3((7++q_-g-l)e(m,2_|!) 



i 

i 

- ^ E 4''e'tanh[/?(m + 7?f + a)] (36) 

i 

Squaring this expression and averaging it over the distribution of patterns, we get 

a^ + {[l^f3iq++q.-q~l)]^}r = l (37) 
Equations (32) and (33) lead to 

Using the transformation Nz^/2a^ = t, we find 

" = C,,, riexp(-.)* = „-|.=r(|, ^) (39) 



The final expression of r is 

where F is the incomplete gamma function defined as 

r{z,p) = / exp{-t)t'-^dt (41) 



Another equation is obtained for m by taking /i = 1 in (31) in the limit /3^oo. This 
result agrees with (11) in section 2. The term P{q+ + q- — q — 1) appearing in (37) 
leads in the limit /3^oo to 

C = 0{q++q.-q-l) 

X|exp(-i!!^) +exp(-(!!i-^) - exp(-^) j (42) 
nar \ ^ 2ar ' ^ 2ar ' ^ 2ar' ] ^ ' 

Finally we have the equations of state as follows. 

CT^ + [(1 - Cf - l]r = 1 (43) 

m = -ert(!!p)-erf(I!l±i)+erf(^) (45) 

VSar v2Q;r ^J2aT 



For simplicity we use a variable fc defined by 
sf^ 1 

2 r(i,^) 
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Equations (43), (44) and (46) are written as 
1 



(1 -cy + k ^^^^ 



J2 



a 



(48) 



For ?7 = 0, one has k = Q and from (43), (44) and (46) one recovers the equations for 
the non-monotonic Hopfield model at T = discussed in the previous section. 

We evaluated equations (45), (46) and (43) and obtained the optimal storage 
capacity ac- We show the results for Uc as a function of ry for the case T = 0, a = 3.0 
in figure 4. A similar result is plotted in figure 5 for the case of T = and a = 1.80. We 
also show the parameter-a dependence of the capacity ac for the cases oi rj ~ 1.0, 0.8 
and 0.6 in figure 6. From this figure we see that as the threshold parameter rj increases, 
more patterns can be embedded by the modified Hebbian rule (30) . It is observed how 
the storage capacity of the non-monotonic Hopfield networks is improved as the value 
of T] increases from 77 = 0. Therefore using the neural networks with non-monotonic 
transfer function and state-dependent synapses, we can get an associative memory 
with the high-quality performance. We also show the overlap m as a function of a for 
the case of = 0.80, a = 3.00 and T = 0, ry = 0.80, a = 1.60 and T = in figure 7 and 
figure 8 respectively. From these figures m(a) is seen to drop to zero discontinuously 
at the critical capacity ac~0.155 for a = 3.0 and ac~0.257 for a = 1.6. 



5. Discussion 



We have investigated the retrieval phase diagrams by the mean field approximation 
in the Hopfield networks with asynchronous dynamics. Mean field approximation 
was extended to the general type of transfer function. The result shows that a 
non-monotonic transfer function yields an enhanced memory capacity for a around 
0.211. This confirms the claim of Morita et al who used the numerical simulation 
for synchronous dynamics and the result of Nishimori and Opris who used the 
equilibrium relation of the Amari and Maginu dynamics for synchronous dynamics. 
The properties of the phase diagram obtained in this paper qualitatively resemble 
those of the phase diagram of the synchronous neural networks. It is interesting that 
our calculation for the asynchronous network also showed the enhancement of the 
capacity as in the synchronous case: The shape of the retrieval phase diagram in this 
paper is similar to that of Nishimori and Opris [|l^ . A difference is that within our 
formulation of the asynchronous networks, the oscillatory phase (limit-cycle phase) 
found by Nishimori and Opris was not obtained clearly. This phase is characterized 
by the behavior of the dynamical order parameter mt which is mt+i > if mt < and 
mt+i < if mt > in the range of < a < 1. As we used the equilibrium statistical 
mechanics to get the phase diagram of the non-monotonic Hopfield model, we can not 
draw definite conclusions about the dynamical order parameter mt- 

For the non-monotonic Hopfield model, the property of asynchronous dynamics is 
an open problem. However, the oscillatory behavior during the process of recursion- 
type solution of equilibrium equations of state may be related to the dynamical 
oscillatory phase found in the same region of the phase diagram. 

We extended our formulation to the general transfer function in this paper. It 
is interesting to investigate whether the storage capacity is enhanced by a transfer 
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function which has a different shape from the stepwise-type one. For the moment, 

wc could not find better transfer functions than the stcpwisc-typc one. And we also 
showed that the Hopfield network with non-monotonic transfer function and state- 
dependent couplings can store large number of patterns. From these results one can 
confirm that the limit of an associative memory with the Hcbb type interactions 
consists in the effects of spurious states each of which has a small correlation with 
embedded patterns. 

Wc may be able to find a new type of network which shows better performance 
than the conventional one by introducing non-monotonic transfer function and state- 
dependent synapses. 

The author thanks Professor Hidetoshi Nishimori for many useful discussions 
during this work. He also thanks Dr. Tomoko Ozeki for many suggestions about 
dynamical properties of Hopfield networks and the Amari-Maginu theory. 
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Figure captions 

Figure 1 

The stepwise-type non-monotonic transfer function. 
Figure 2 

Phase diagram for the non-monotonic transfer function in figure 1 obtained by the 
mean field approximation. Retrieval is successful in the region R (normal retrieval) 
and unsuccessful in N/R. ac{a = oo) = 0.138 (consistent with Amit et al) and 
ac{a = 1.77) = 0.211 (maximum value). 
Figure 3 

The a-T phase diagram of the Hopfield model with non-monotonic transfer function 
(a = 1.80). Retrieval is successful in R (retrieval phase) and unsuccessful in SG (spin 
glass phase). 
Figure 4 

A slice of the phase space for the non-monotonic networks with threshold 77 at T = 
and a = 3.0. The Hopfield value ac = 0.138 is found at 77 = 0; for a threshold 77 equal 
to 1 the optimal capacity increases to ac = 0.171. 
Figure 5 

A slice of the phase space for the non-monotonic networks with threshold r/ at T = 
and a = 1.8. 
Figure 6 

Optimal storage capacity ac{a) of the non-monotonic network with thresholds r] = 
1.0,0.8 and 0.6 at T = 0. 
Figure 7 

The order parameter m{a) at T = and 77 = 0.80 and a = 3.00. 
Figure 8 

The order parameter 771(a) at T = and 77 = 0.80 and a = 1.60. 



